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SUMMARY 
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Background: 


Why this study? This study provides New Mexico’s policymakers and practitioners with an evaluation of the 
performance of charter schools (with a special focus on those employing a school-wide dual language model) using data 
from the state’s students. 


Which schools were in the study sample? Public charter schools in the state that used a lottery to admit students for 
school years 2009-10 to 2017-18 were eligible for this study (21 schools ultimately participated, which represents 70% 
of the eligible schools). 


How was this study conducted? Our implementation study surveyed principals from charter schools and conducted 
virtual site visits to a subset of dual language charter schools to understand the key ingredients of an education at the 
schools that participated in our study. Our impact study used (1) a lottery study, the most rigorous analysis available to 
evaluate charter school effectiveness, and (2) a matching study to examine the academic effects of multi-year attendance 
at a charter school. 


Who conducted this study? A team of researchers from Abt Associates worked closely with two non-profit 
organizations in New Mexico — Public Charter Schools of New Mexico and Dual Language Education of New Mexico — 
to conduct this study. The New Mexico Public Education Department provided critical student data for this study. 


Key Findings: 
Charter Schools in the Study (lmplementation Study): 


e Charter schools in the study were founded individually by teachers, parents, and/or other community 
members to respond to a perceived community need and/or demand from families for local public schools 
offering high-quality, rigorous education. The local derivation of these schools has resulted in schools with 
a diversity of missions, strategies, and key educational components. The schools are located across the 
state, with slightly fewer than half located in small towns and rural areas. 

e Charter schools in this study served fewer economically disadvantaged students and more White students 
compared to all charters in the state (to some extent) and to all public schools in New Mexico (to a 
greater extent). This could be due in part to school location and other logistical barriers such as lack of 
buses preventing economically disadvantaged families from enrolling. We provide recommendations for 
expanding equity of access below. 


Charter School Performance (Impact Studies): 

e Our charter school lottery study included 10 schools that had well-documented records about their 
initial spring lotteries. The study found no statistically significant impacts on measured student academic 
outcomes for elementary-aged and middle school-aged students, nor on college enrollment for high 
school-aged students who were offered admission in the initial lottery compared to those students not 
offered admission. 


Our charter school matching study included all 21 schools and focused on the effect of continual 
attendance at a charter school versus continual attendance at a non-charter school. The results were 
mixed, suggesting high school charters increased college enrollment and elementary charters reduced 
mathematics achievement, relative to comparison students. 


Together these findings suggest that the locally established charter schools in our sample will not remedy the lower 
performance of students in New Mexico relative to the nation nor close the gaps in educational outcomes between 
more and less privileged students in the state. However, the findings for college enrollment indicate that several charter 
schools in our sample have effectively identified mechanisms for ensuring more students enroll in college. 
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Dual Language School Performance (Impact Study): 

e Our dual language matching study focused on the effect on students’ English Language Arts (ELA) 
performance of five years of dual language education at a charter school compared to five years of standard 
instruction at a non-dual language school. All of the dual language charter schools and comparison schools 
in the study were located in the Albuquerque area. We did not find any statistically significant differences 
in English Language Arts (ELA) achievement in fifth grade relative to comparison students. Our findings do 
however suggest that the sub-group of English learners who continuously attend a school-wide dual 
language charter school perform similarly to or better in ELA than English learners who continuously 
attend non-dual language schools. However, approximately half of all English learners who enrolled in the 
focal schools over the follow-up period were enrolled for two school years or less (48 percent) so these 
findings do not apply to the typical English learner in these dual language schools. 


The study found descriptive evidence from principal reports that dual language students are developing 
proficiency in Spanish. Principals were asked about growth in Spanish proficiency for their students and 
reported dramatic growth in proficiency in oral and written Spanish at grade level in the period from 
enrollment in kindergarten to fifth grade. Although we cannot say what principals in the comparison 
schools would report about growth in the Spanish language skills, the gains reported by dual language 
principals suggest they are achieving their goal of developing students who are bilingual and biliterate. 


Recommendations for New Mexico’s Policymakers 


|. Support quality instruction and programming at locally designed charter schools. 
¥ invest in outcomes analyses that encompasses the full range of hypothesized effects on students (described in the 
charter school theory of change in this document); 
¥Y consider ways to support high-quality instruction for charter school students. 


Improve equity of access to charters. If New Mexico has the goal of making charter schools accessible to all 

students in the state, the state could look at ways to remove barriers: 

vY increase awareness of charter schools’ existence and ensure all parents (especially non-English speakers) know 
schools are tuition free; 


invest resources necessary to offer busses to avoid transportation barriers; 

consider the location of new charters and prioritize census tracts where economically disadvantaged and minority 
students reside — and offer support to community groups focused on education of disadvantaged populations 
looking to start new charter schools; 


consider changing state regulations to allow charter schools to hold separate lotteries for subgroups of students to 
ensure admission of disadvantaged students. 


Increase access to dual language programs. 

Y in addition to the above, consider changing state regulations to allow or require dual language charter and magnet 
schools to offer admissions priority to English learners (or hold separate lotteries for Spanish and English dominant 
speakers); 


Y to the extent that the state has the goal of increasing linguistic equity in the state’s schools while supporting English 
learners’ achievement, consider expansion of quality dual language programs. 


Consider standardizing and centralizing charter lottery data systems. 


vY Efforts to improve the lottery tracking/data systems would help support compliance with charter lottery 
requirements and would help support future rigorous research. 


Revamp data systems and data polices to support linguistic equity. If New Mexico is to fully measure 

students’ academic, linguistic, and socio-cultural capabilities, the state could bolster its data systems accordingly by: 

Y Giving assessment data in non-English languages equal priority in all data systems to ensure students’ capabilities are 
honored and data are available to assess related programs and policies. 


Lastly, we recommend additional research that (1) explores the approaches at the participating charter high schools 
that appear to increase college enrollment; (2) expands the number of charter and dual language school students 
included in the research effort; (3) identifies individual charter schools that are effective and their key elements and (4) 
further aligns with the respective program’s theory of change logic model in terms of outcomes explored (ie. student 
engagement and family empowerment for charter schools, and partner language written and oral proficiency and 
socio-cultural competence for dual language programs). 


INTRODUCTION 
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The goal of the New Mexico Charter School Study is to assist New Mexico 
policymakers and practitioners in understanding the performance of charter schools in 
the state. About 7% of New Mexico’s public school students attend one of 97 charter 
schools, which is a similar proportion to the United States overall.' Charter schools are 
seen by proponents as a way to offer students and families more choice and educational 
opportunities. Further, proponents argue that charters could offer higher quality 
education because the schools are free from most of the traditional bureaucracy and 
regulations that divert educator energy and resources. On the other hand, charter 
school opponents argue that they distort the distribution of resources in the public 
education system and risk skimming the children of parents who would be more likely 
to be engaged in their traditional public school. As charter schools play a sizeable role 
in New Mexico’s public education system, it is important to know how they are 
performing. 


Prior studies of charter schools have focused on different types of charter schools than 
those in New Mexico. For example, many prior research studies elsewhere have 
focused on charters in urban and suburban areas or on charters run by charter 
management organizations (CMOs). New Mexico’s charter schools are often in rural 
areas, and the state does not have any charter schools run by CMOs; instead, charter 
schools in New Mexico are grass roots institutions run by independent governing 
councils. 


In addition to studying charter schools more generally, this study has a special focus on 
elementary charter schools that employ a school-wide dual language education model. 
Dual language education provides instruction in literacy and content in two languages. It 
has been shown to be effective at improving students’ academic achievement," and it is 
an especially timely research topic for New Mexico given the emphasis of the Yazzie- 
Martinez lawsuit on ensuring the provision of equitable bilingual and multicultural 
education." 


The remainder of this document presents what we learned from this study. We 
encourage the reader to review the entire report, but those interested in specific Location of 21 Study Schools 


chapters can skip directly to them using the hyperlinks below — , 

UAL 
LANGUAGE 
SCHOOLS 


INCLUDES | 
e Charter School Findings presents findings about the performance of charter 


schools by grade levels of the students served (high school, middle school, 
and elementary school). 

e Dual Language Charter School Findings, presents all findings from analyses s 
related to dual language education, including (1) Spanish proficiency © ; 
reported by principals, (2) the impact of attending a dual language school 2] u 
on all students and on English language learners specifically, and (3) findings ROnratean s 
from virtual site visits conducted by Dual Language Education of New 0... 
Mexico. | } 

e Schools in the Study describes the key traits of the schools we studied. Data 
come from a survey of school principals (which had a 95% response rate). , 
In this chapter, we summarize the origin of the focal schools, their missions, @ 3 (1) 
their instructional time, and other key ingredients of the school. i SOUTHEAST | 

J e Who Is Served by Charter Schools in New Mexico presents the 2S . 


NORTH | 
CENTRAL 


NORTHEAST 


SOUTHWEST // 


demographic traits of students attending the 21 schools that are part of this 
study, as well as students in all 97 charter schools and all 862 public schools 
in the state. 
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INTRODUCTION 


e Theory of Change Logic Models documents the theorized pathways to 
improved outcomes behind charter schools and dual language schools. 
These logic models were created by study partners from Public Charter 
Schools of New Mexico and Dual Language Education of New Mexico in 
collaboration with the Abt Associates team.' Our study is not able to 
report on all the components of these logic models, but we believe they 
are critical tools for understanding how each model can affect schools and 
communities and for supporting existing programs. 

e We have also created a separate Technical Appendix for a research 
audience. 


Implementation Study Design and Questions. We conducted an implementation 
study that used surveys of principals from all charter schools and virtual site visits to 
dual language schools. Our implementation study aimed to understand the key 
ingredients of an education at the charter schools that participated in our study. 


Impact Study Research Design and Questions. We answered seven research 
questions with seven separate impact sub-studies. Exhibit | (below) presents the key 
characteristics of each sub-study, including the topic, the rigor of the study design, the 
research priority (meaning the level of importance -primary or secondary- within the 
study), and the number of schools and students included in the sub-study. “ For the 
reader’s ease, we repeat these key characteristics when presenting findings in later 
exhibits. The full research questions are included in the chapters presenting findings, 
along with results. 


Exhibit 1. Key Characteristics of Sub-Studies 


Number of Focal Number of All 
Schools in Study Students in Study 


Research 
Priority 


Rigor of 


High School 
Charter Lottery 


Middle School 
Charter Lottery 


iff 


Elementary 


3 Charter Lottery 
High School erenoreteres 
Aver a ee 
Matching vitae 
PIX) 
Middle School ee isereeee 
ROT @tOre) 
5 as =f | secondary | sive ehe aferry it mn 
Matching i) 1 .) 
Elementary . AE 
(3 Charter f sei 
uae 
Matching 


Dual Language us 
7 School Matching we T secondary 


| Because our partners, Public Charter Schools of New Mexico and Dual Language Education 
of New Mexico, have a vested interest in the results of the lottery and matching studies, 
they were involved only in the conduct of the implementation studies; they did not have 
access to nor contribute to lottery and matching studies’ data collection or analyses. 
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i 


A lottery study with low attrition is 
the gold standard research design for 
showing that the treatment caused the 
outcome (golden ribbon for rigor) 


A lottery study with high attrition 
uses the gold standard design but is 
less rigorous due to sample loss (silver 
ribbon for rigor) 


A matching study with baseline 
equivalence is a less rigorous design 
that attempts to mimic lottery results 
(bronze ribbon for rigor) 


Findings from a pre-specified analytic 
approach designated as answering 
primary research questions 


Findings from a pre-specified analytic 
approach designated as answering 


additional secondary research questions 


Number of focal schools included in 
the study (1 building = 5 schools) 


Number of students included in the 
study (1 person = 100 students) 
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INTRODUCTION 


We used two research designs to answer our key questions — a lottery design and a 
matching design. Together these two designs present a balanced and nuanced view of 
the effects of charter schools in New Mexico because the lottery study does not require 
enrollment at the school whereas the matching study examines continuous enrollment. 


LOTTERY STUDY DESIGN. The lottery sub-studies’ use the most rigorous analysis 
available to evaluate charter school effectiveness. The three lottery sub-studies 
test the effects of students being offered admission to a charter school on the students’ 
math and English Language Arts (ELA) achievement, looking at effects separately for 
elementary, middle, and high school charter schools. These studies are limited to 
charter schools that are oversubscribed, use lotteries that randomly allocate seats 
to students, and had sufficient records available about their initial lotteries.” 


The lottery ensures that the students offered admission to a charter school (the 
“treatment” group) and the students not offered admission (the “control” group) have 
the same backgrounds and characteristics, including characteristics that cannot be 
measured. As a result, the comparison is “apples-to-apples” and any differences in their 
achievement or college enrollment can be attributed to the opportunity to attend the 
charter school; that is, being offered a admission through the lottery. This design allows 
us to make causal statements about the effect of being offered admission to a charter school. 


MATCHING STUDY DESIGN. The matching sub-studies” test the effects of multi-year 
attendance in charter schools and charter/magnet dual language schools on students’ 
math and ELA achievement. As in the lottery sub-studies, in the matching sub-studies 
we again include only oversubscribed schools, and we look at effects separately for 
students attending elementary, middle, and high schools. 


This type of study is termed quasi-experimental because students being compared were 
not randomly assigned to the charters. Instead, students who chose to attend and 
remain in a charter school over multiple years are compared to similar students who 
attended a single non-charter traditional public school. The study matches each student 
in a charter school to a student in a non-charter on their prior achievement, 
race/ethnicity, English learner status, and Free or Reduced Price Lunch (FRPL) eligibility 
(if available), but students may differ in several ways that cannot be measured. As a 
result, this may be more of an “apples-to-oranges” comparison and any differences in 
student outcomes cannot be attributed solely to charter attendance; instead, 
differences might reflect charter attendance and/or they might come from differences 
between students such as differences in parental education, family support, or 
motivation)). 


Although matching studies do not permit us to make the same causal statements we 
can make in lottery studies (because matching studies do not require students to be 
randomly assigned), matching studies allow us to include substantially more schools. 


Finally, this study focuses on the time period from about 2009-10 
PRE- through 2018-19, before the COVID-I9 pandemic. Because of the 
massive shifts in educational delivery during the pandemic, the results 
ra are best applied to time periods when students are educated primarily 
in school buildings. 


2 Ten participating schools provided lottery records that allowed us to conduct the experiment: 
i.e. they allowed us to (I) identify the students that were admitted in the initial lottery and 
(2) follow the students from application through outcome measurement. 
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Research Terminology 


Oversubscribed in our study 
means having had at least 15-20 students 
on a wait list after the initial lottery in at 
least one grade. 


A LOTTERY STuby is the 
most rigorous research 
design. 


Our MATCHING STUDY 
measures the effect of 
continuous attendance. 
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CHARTER SCHOOL FINDINGS 


Charter School Findings 


Lottery Study Findings 


The charter school lottery studies were designed to answer three research questions 
about the impact of oversubscribed elementary, middle, and high school charters”" on 
key academic outcomes: 


|. What is the impact of oversubscribed elementary charter schools in New 
Mexico on grade 3 math and English Language Arts (ELA) achievement? 


2. What is the impact of oversubscribed charter middle schools in New Mexico 
on grade 8 math and ELA achievement? 


3. What is the impact of oversubscribed charter high schools in New Mexico on 
college enrollment? 


These sub-studies prioritize rigor and allow us to say whether these charter schools 
caused any statistically significant impacts to be observed between the 
treatment group and control group when the error bands do not cross zero 
(i.e. when we can be reasonably confident that the difference in outcomes is not due to 
chance). However, the strict requirements of lottery studies mean we could only 
include a limited number of schools in these analyses (as shown in the Design at a Glance 
summaries throughout this chapter). 


Across all three types of schools, the lottery sub-studies did not find evidence that 
charter schools improved academic outcomes. The wide error bands around all the 
primary impact estimates suggest that admission to a charter school could either reduce 
outcomes or could improve outcomes. 


Lottery studies rigorously estimate the difference in outcomes for students who applied 
to the focal schools in the study. However, the focal schools in the study are not 
representative of charters schools in New Mexico generally. See the chapter Who is 
Served by Charters Schools in New Mexico for a discussion comparing our overall 
study sample of oversubscribed charter schools to all charter schools in the state and 
to all schools in the state. 


These analyses focus on a limited set of student academic outcomes. It is possible that 
these charter schools improve other outcomes that we did not measure such as teacher 
retention, and other elements of the theory of change for charter schools (see the 
charter school theory of change logic model in Exhibit 18 at the end of this report). 
Other rigorous research has found statistically significant effects on both student and 
parental satisfaction, which we were unable to measure in this study due to resource 
limitations." 


At the high school level, the proportion of students who 
enrolled in college in the fall after expected high school 
graduation did not differ significantly between students 
offered admission at a charter school and those not offered 
admission. 


e The rate of college enrollment for treatment group students was I5 
percentage point higher than for control group students. Wide error bands 
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Research Terminology 


Statistically significant 
means believed to be a real difference in 
the population. 


The treatment group includes 
students offered seats at a focal charter 
school the initial spring lottery. 


The control group isa term used 
in lottery studies only, and includes those 
students not offered admission at a focal 
charter school during this same lottery. 
Our treatment and control students are 
virtually the same, except that some were 
offered admission and some were not. 


Error bands show the range of 
plausible values when accounting for 
sampling error. 


Lottery Study Steps: 


Charter school’s initial 
(first) lottery is conducted 


Students offered a seat are 
Treatment Students 


Students NOT offered a 
seat are Control Students 


Students are dropped from 
the study if they don’t have 
outcome data available 
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CHARTER SCHOOL 


indicate that moderate negative effects and very large positive effects are 
plausible.’ 

e Two exploratory analyses show no difference in college persistence or 
retention.’ Persistence is enrollment in the fall after expected high school 
graduation and in the following fall. Retention is enrollment in the same college 
in the fall after on-time graduation and the following fall. These exploratory 
analyses support the interpretation that charter high schools have not been 
shown to improve college outcomes. 


Exhibit 2. High School Lottery Study Findings: Difference Between 
Treatment (Offered Admission) and Control (Not Offered 
Admission) 


Panel |: Design at a Glance 


High School 
Charter Lottery 


Panel 2 Findings 


FINDINGS 


fo tigi 


50 $3 

a ie 

30 Bae 
5 15.12 = Ne: | | 
& 0 @ 0.73 © a. I The line represents sampling error 
5 -10 = estimate from our data (not 
5-20 3 5 statistically significant) 

30 ERe 

50 35 

College Enrollment College Persistence College Retention > 
(Primary) (Exporatory) (Exploratory) 


Source(s): National Study Clearinghouse (NSC) and NM PED Administrative Data 

Sample: The NSC analysis sample includes 684 (401 treatment and 283 control) students. 

The NM PED analysis sample for High School Graduation and ELA achievement outcomes include 638 students (267 
treatment and 371 control). The NM PED analysis sample for the Algebra Il outcome includes 480 (181 treatment and 
299 control) students. 

Notes: Statistical significance levels for two-sided tests are indicated with asterisks, as follows: *** = 1 percent; ** = 5 
percent; * = 10 percent. None of the impacts are significantly different than zero. All primary and exploratory comparisons 
in the table are experimental. Students in the sample were not enrolled are included with 0 values. 


3 Admission to a charter high school could reduce college enrollment by up to 8 percentage 
points or increase college enrollment by up to 38 percentage points. 


4 Persistence rates are about the same between the treatment and control groups (40 vs. 39 
percent), as are the retention rates (35 vs 33 percent). Error bands indicate that moderate 
negative or large positive effects are plausible. 
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CHARTER SCHOOL FINDINGS 


At the middle school level, neither 8°" grade ELA nor math 
achievement differed significantly between students offered 
admission at a charter school and those not offered 
admission. 

We find no evidence that admission to the eight charter middle schools significantly 


increased 8" grade ELA achievement or the proportion of students passing the Algebra 
| PARCC test. 


e About the same proportion of students took and passed the Algebra I 
PARCC test in each group (24 vs 23 percent). The error bands indicate that 
large positive and negative effects are plausible.° 

e Average 8 grade ELA achievement scores were slightly lower for the 
treatment group than for the control group (difference is less than a tenth of 
typical annual gains in reading), with error bands including both large negative 
effects and moderate positive effects.° 

e Anexploratory analysis of 7 grade science achievement finds no difference 
in average outcomes between students offered admission and students not 
offered admission. 


Exhibit 3. Middle School Lottery Study Findings: Difference Between 
Treatment (Offered Admission) and Control (Not Offered 
Admission) 


Panel 1: Design at a Glance 


as OV la 


—— 


Panel 2: Findings 


o 0.400 50 
E 30 2 
= 0.200 0.134 i Key: 
2 1.7 0's 
& 0.000 -0.012 @ : oe I The line represents sampling error 
i 10 
 -0.200 30 5 @ estimate from our data (not 
s a statistically significant) 
5 -0.400 50 
Yn 
8th Grade English 7th Grade Science Passed Algebra | or 
Language Arts (Exploratory) Higher 
(Primary) (Primary) 


Source: NM PED Administrative Data 

Sample: The ELA analyses include 1,650 (321 treatment and 387 control) students. The Algebra | analysis includes 
1,585 (304 treatment and 1,281 control) students. Students who took the 8 grade math PARCC test (Pre-Algebra) are 
included in the analysis as students who did not take and pass the Algebra | test. Students who took a higher-level math 
PARCC tests (Geometry or Algebra II) are included in the analysis as students who took and passed the Algebra | 
PARCC tests. The Science analysis includes 1,551 (300 treatment and 1,251 control) students. 


5 Admission to a middle school charter could reduce the probability of passing Algebra | by up 
to 6 percentage points or increase the probability of passing by up to 9 percentage points. 


6 Middle school charters could reduce 8t grade ELA achievement by more than half of typical 
annual gains or increase ELA achievement by about two-fifths of typical annual gains. 
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CHARTER SCHOOL FINDINGS 


Notes: After correcting for multiple comparisons, the 90% confidence interval for ELA achievement is (-0.151, 0.127) and 
the 90% confidence interval for Algebra Il is (-7.2, 10.7). See Appendix for additional detail. 


At the elementary school level, the 3rd grade math and ELA 
achievement of students offered admission at a charter 
school did not differ significantly from that students not 
offered admission. 


We find no evidence that admission to the four elementary charters in the study 
significantly increased 3rd grade achievement in math or ELA. 


e Estimates show higher 3™ grade ELA test scores for students admitted to 
the focal schools (equivalent to one-fifth of typical annual gains in reading); 
however, the wide error bands around the estimates indicate that small 
negative effects or large positive effects are plausible.” 

e The estimated difference in 3" grade math scores is small (less than one- 
tenth of the typical annual gain in math) and error bands indicate that small 
negative or moderate positive effects are plausible. ° 

e An exploratory analysis of 4° grade science achievement finds no 
difference in average outcomes between students offered admission and 
students not offered admission. 


Exhibit 4. Elementary School Lottery Study Findings: Difference 
Between Treatment (Offered Admission) and Control (Not Offered 
Admission) 
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Source: NM PED Administrative Data 

Sample: Math and ELA analyses include 452 (115 treatment and 337 control) students. Science analysis includes 362 
(89 treatment and 273 control) students. 

Notes: After correcting for multiple comparisons, the 90% confidence interval for ELA achievement is (-0.163, 0.404) and 
the 90% confidence interval for math achievement is (-0.166, 0.292). See Appendix for additional detail. 


7 Admission could reduce 34 grade ELA scores by as much as one-fifth of typical annual gains or 
increase test scores by slightly more than one half of typical annual gains. 


8 Admission could reduce 3"4 grade math scores by about one-sixth of typical annual gains or 
increase math scores by just over one-fourth of typical annual gains. 
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Matching Study Findings 


The charter school matching sub-studies were designed to answer three research 
questions about the effects of continuous attendance at a focal school on key academic 
outcomes: 


|. What is the impact of enrollment in first through third grade at oversubscribed 
elementary charter schools in New Mexico on grade 3 math and ELA 
achievement? 


2. What is the impact of enrollment in sixth through eighth’ grade at oversubscribed 
charter middle schools in New Mexico on grade 8 math and ELA achievement? 


3. What is the impact of enrollment in ninth and tenth grade'® at oversubscribed 
charter high schools in New Mexico on college enrollment? 


These sub-studies seek to compare each focal charter school to the traditional public 
schools that that would be the most likely alternative schools for students. School 
lottery records allow us to identify traditional public schools where applicants often 
enroll. Enrollment records allow us to identify students who switch into or out of the 
charter school and the traditional public schools they attend. The matched comparison 
sample is drawn from students enrolled in these traditional public schools. 


These sub-studies prioritize learning about the effect of charter schools on students 
who chose to remain enrolled in a charter school during all grades served by the school 
and can therefore be expected to have received the full benefit of the educational 
program. These students are compared to students who were enrolled in a single, 
traditional public school over the same grades. This focus on students who did not 
switch schools means that the analysis excludes students whose schooling is disrupted 
by one or more moves. 


Although they are less rigorous than the lottery sub-studies, the matching design allows 
us to include a larger number of schools and students in the analyses (as shown in the 
Design at a Glance summaries throughout this chapter) reducing the width of the error 
bands and increasing the range of schools to which our findings would apply. 


The matching study found mixed impacts on educational achievement or attainment, 
with , positive findings at the high school level for college enrollment, negative findings 
at the elementary level for mathematic, sand otherwise null (no difference) findings 
relative to the comparison group. 


A higher proportion of students who attended charter high 
schools in 9° and 10° grade enrolled in college in the fall 
after expected high school graduation than similar students 
enrolled in traditional public schools in 9" and 10‘ grade. 


This finding suggests that enrollment in charter high schools may increase college 
enrollment. 


e The college enrollment rate is 6 percentage points higher for students 
enrolled in charter high schools than for students enrolled in traditional high 
schools. As shown in Exhibit 6, this impact is larger than the difference among 
urban, suburban, and rural schools (4 percentage points) but not as large as 
the difference between low-income and higher-income schools (15 percentage 


9 Seventh through eighth grade in the one school that doesn’t have a sixth grade. 


10 Tenth grade only at the two schools that don’t have ninth grade. 
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Matching Study Steps: 


Treatment schools are 
the 21 focal schools in 


our study 


Comparison schools 
commonly share 
applicants/students with 
treatment schools 


Each treatment school 
student is matched with a 
comparison school student 
on race/ethnicity, FRPL 
status, EL status, and 
baseline test score 


Treatment students are 
excluded from the study if 
a comparison student 

fra icelaMor-lalatolel oX-M Collate! 


Research Terminology 


The comparison group is a 
term used in matching studies, and 


includes the students from non-charter 


schools who were matched to the 
students in charter schools on prior 


achievement, race/ethnicity, FRPL status, 


and English learner status. 
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points) or the difference between high-minority schools and low-minority 
schools (I | percentage points). 


Although the sample includes early college high schools, the estimated impact is not 
associated with dual enrollment, where students enroll in community college 
classes while in high school,'' or of the requirement that students be enrolled in 
both 9 and 10" grade.'*'? This finding is consistent with our lottery study findings. 
Focusing on the four charter high schools included in the lottery sample, yielded an 
impact estimate similar to the lottery study and the error bands fall within with 
error bands of lottery study estimates. '* 


Exhibit 5. High School Matching Study Findings: Difference Between 
Treatment (Charter) and Comparison (Non-Charter) 
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Source(s): NSC Data 

Sample: Sample includes 1,542 (771 treatment and 771 comparison) students. 

Notes: Statistical significance levels for two-sided tests are indicated with asterisks, as follows: *** = 1 percent; ** = 5 
percent; * = 10 percent. 


'! Given that there are early college high schools in the sample, observed college enrollment 
could be dual enrollment if high school students who did not graduate on time. That is not 
where the impact estimate comes from. 


12 We estimated the difference excluding schools with dual enrollment programs and found a 
7-percentage point difference between students enrolled in charter schools and students 
enrolled in traditional public schools. 


'3, We estimated the difference between students enrolled in charter schools in 9t grade and 
similar students enrolled in traditional public schools in 9t grade and found a difference in 
college enrollment of 8 percentage points. 


14 College enrollment among students enrolled in charter schools in 9th grade is 13 percentage 
points higher than for similar students enrolled in traditional public schools in 9t grade. The 
error bands around the estimate range from a 2 percentage point increase to a 24 
percentage point difference. 
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Exhibit 6. National College Enrollment in the First Fall After High 
School Graduation, by School Type, Compared to This Study’s 
Findings 
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Source: National Student Clearinghouse Benchmark Report 2017-18; Study findings. 


Math and ELA achievement did not differ significantly 
between students enrolled in charter schools from 6" to 8" 
grade and similar students enrolled in traditional public 
schools from 6" to 8 grade. These analyses offer no 
evidence that enrollment in charter schools from sixth to 
eighth grade significantly increases or decreases 
achievement in math or ELA compared to enrollment in a 
traditional public school from sixth to eighth grade. 
e Average 8" grade ELA test scores were slightly higher among students 

enrolled in charter schools (about two-fifths of typical annual gains in 

ELA), with error bands including both small negative and large positive 

effects. '° 


e Although a lower proportion of students enrolled in the charter schools 
took and passed the Algebra | PARCC test (15 percent vs. 19 
percent), the error bands around the estimate indicate that large negative 
or small positive impacts are plausible. '* 


15 Middle school charter enrollment could reduce ELA achievement by less than a tenth of 
typical annual gains or could increase ELA achievement by almost a full year of typical annual 
gains. 


16 Middle school charter enrollment could be associated with a reduction in the proportion of 
students passing Algebra | as large as || percentage points or could be associated with an 
increase up to 2 percentage points. 
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Exhibit 7. Middle School Matching Study Findings: Difference 
Between Treatment (Charter) and Comparison (Non-Charter) 
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Source(s): NM PED Administrative Data 

Sample: The sample includes 480 (248 treatment and 248 comparison) students. Students who took the 8" grade math 
PARCC test (Pre-Algebra) are included in the analysis as students who did not take and pass the Algebra | test. Students 
who took a higher-level math PARCC tests (Geometry or Algebra II) are included in the analysis as students who took 
and passed the Algebra | PARCC tests. 

Notes: Statistical significance levels for two-sided tests are indicated with asterisks, as follows: *** = 1 percent; ** = 5 
percent; * = 10 percent. After correcting for multiple comparisons, the 90 percent confidence interval for ELA 
achievement is (-0.231, 0.109) and for math achievement is (-0.379, -0.025). 


Students enrolled in elementary charter schools from 1st to 
3rd grade had significantly lower 3rd grade math test scores 
than similar students enrolled in traditional public schools 
from Ist to 3rd grade. We did not find a significant difference 
in ELA scores between the groups. 


Secondary analyses suggest that enrollment in elementary charter schools from first to 
third grade may reduce academic achievement compared to enrollment in a traditional 
public school from first to third grade. 


e The average student enrolled in elementary charters from first to third grade 
scored significantly lower on 3"? grade math tests than matched comparison 
students (difference equivalent to one-fifth of typical annual gains in math). 

e Although estimates indicate that 3"? grade ELA scores are also lower for 
students in elementary charters than comparison students (difference is about 
one-tenth of typical annual gains in ELA), error bands include large negative 
effects and small positive effects. '” 


'7 Our findings suggest that enrollment in elementary charters could reduce ELA test scores by 
more than one-half of typical annual gains or could increase ELA scores by about one-eighth 
of typical annual gains. 
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The observed difference in math outcomes is not due to differences in demographics 
nor differences in kindergarten reading levels.'® We cannot rule out the possibility that 
kindergarten math levels differed between the groups or that the difference in math 
outcomes is a consequence of another unobserved difference between the groups. 


Exhibit 8. Elementary School Matching Study Findings: Difference 
Between Treatment (Charter) and Comparison (Non-Charter) 
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Source(s): NM PED Administrative Data 

Sample: The sample includes 480 (248 treatment and 248 comparison) students. 

Notes: Statistical significance levels for two-sided tests are indicated with asterisks, as follows: *** = 1 percent; ** = 5 
percent; * = 10 percent. After correcting for multiple comparisons, the 90 percent confidence interval for ELA 
achievement is (-0.231, 0.109) and for math achievement is (-0.379, -0.025). 


'8 Even before matching, the students enrolled in the focal schools from first to third grade 
were similar to students enrolled in the comparison schools from first to third grade, both 
in terms of demographics and kindergarten reading level (55t percentile vs 53'4 percentile). 
After matching, the average charter student and the average matched comparison student 
both scored at the 55" percentile on the kindergarten reading test. 
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In this chapter we report on the dual language charter/magnet'? elementary school 
study findings. Dual language education is a model that provides instruction in literacy 
and content in two languages, with the goal of student bilingualism, biliteracy, and 
sociocultural competence. This chapter is structured differently from the previous 
chapter on charter schools because we provide context from our principal surveys and 
virtual site visits to dual language schools conducted in partnership with Dual Language 
Education of New Mexico. 


Although we could not directly measure Spanish proficiency in the dual language 
matching study, we were able to ask DL school principals to use assessment data 
whenever possible to provide information on their students’ Spanish proficiency in both 
oral and written Spanish. Principals reported the percentage of students who entered 
kindergarten proficient in oral and written Spanish at grade level and the percentage 
who, after continuous enrollment, were proficient at the end of fifth grade at grade 
level. Exhibit 9 reports the results of our query. 


Exhibit 9. Principal-Reported Oral and Written Spanish Proficiency 
Increased Dramatically from Kindergarten Entry to Fifth-Grade Exit 
for both native Spanish and native English speakers with 
Continuous Enrollment at a Dual Language School 


Principals reported dramatic 
growth in oral and written 
Spanish proficiency at grade level 
from Kindergarten to fifth grade 
with continuous DL enrollment. 


5% 


Oral Spanish Written Spanish 


@ Kindergarteners proficient at grade level 5th Graders proficient at grade level 


Source(s): Email Exchanges with DL Principals 
Sample: The sample includes 4 of the 5 DL schools that participated in the study. 


The average principal reported that while only about 15% of Kindergarteners were 
proficient at grade level in oral Spanish, 75% were in fifth grade. The growth was 
similarly dramatic for written Spanish, increasing from approximately 5% in 
Kindergarten to approximately 64% in fifth grade. 


Although we cannot say what principals in the comparison schools would report about 
growth in the Spanish language skills, the gains reported by dual language principals 
suggest these dual language schools are generally ensuring the majority of their students 
are bilingual and biliterate. 


'9 We include four oversubscribed charter schools and one oversubscribed magnet school. 
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Dual Language Matching Study Findings 


The dual language matching study measured the effects of continuous attendance at a 
focal DL school. We answer one secondary questions about dual language schools: 

|. What is the impact of enrollment in K or |* to 5th grade at oversubscribed dual 
language charter/magnet schools in the Albuquerque/Rio Rancho area on grade 5 
English Language Arts (ELA) achievement? 


Similar to the charter matching study, the dual language matching study seeks to 
compare dual language schools to non-dual language schools that parents consider 
alternative options to enrolling their child in dual language schools. School lottery 
records allow us to identify public schools where applicants often enroll. Enrollment 
records allow us to identify students who switch into or out of dual language school 
and the English language schools they attend. The matched comparison sample is drawn 
from students enrolled in these English language schools. 


This study focuses on understanding the effect of dual language schools on students 
expected to receive the full benefit of the program, those enrolled from first to fifth 
grade. These students are compared to students who were enrolled in a single, English 
language school over the same key grades. We committed to this analytic approach in 
our pre-analysis plan because prior studies of dual language found positive effects in 5" 
grade and because experts on dual language told us that they would not expect to see 
impacts for students who were not enrolled for the five years. 


Below, we report these results separately for our overall sample of students, and for 
English learners separately (as they are a key group of interest in bilingual education 
programs). 


Overall Sample 

The overall sample for the DL matching study is very small due to assessment data being 
unavailable.”° The analysis only includes 90 students who were continuously enrolled in 
a dual language school. 


Due to small sample sizes and variation across students, 

estimates of the difference in 5" grade ELA achievement 

between dual language students and students in English 

language schools are too uncertain to support interpretation. 

e Although the average ELA test scores are higher for students enrolled in 

focal DL schools from |* to 5" grade (equivalent to about half of typical annual 
gains in ELA), the error bands include both large negative and large positive 
effects.*| These error bands are so wide that no conclusions can be drawn 
from the analysis. 


20 Although the NM PED required that kindergarten and first graders take a reading test 
starting in the 2013-2014 school year, most dual language schools fielded the Spanish version 
of the test and those scores are not available in the NM PED school accountability data 
system. Because matching requires a test score, the sample of students never identified as 
English learners is limited to the students for whom we observe a first grade English language 
test score. We are able to expand the sample slightly by using the first grade ACCESS score, 
an assessment tracking progress for English learners, to match English learners 


21 Our estimates suggest that dual language could decrease ELA scores by three-quarters of 
typical annual gains or increase ELA scores by one-and-three-quarters of typical annual gains. 
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Exhibit 10. Dual Language Elementary School Matching Study 
Findings: Difference Between Treatment (Dual Language) and 
Comparison (Non-Dual Language) 


All Students 


Panel |: Design at a Glance 


posi, OF te fr 


Panel 2: Findings 


3 
€ec2 
0.700 3 Bo 
= 8 wo 
2 0.500 8s 
5 z Key: 
= 0.300 ; 
° 
% 0.100 0.183 I The line represents sampling error 
2 0: no difference 
QO -0.100 
= @ estimate from our data (not 
z 0.300 5s statistically significant) 
7 652 
A -0.500 gee 
Leo 
-0.700 EE 
5th Grade English Language Arts ae 


(Secondary) 


Source(s): NM PED Administrative Data 

Sample: The sample includes 180 students (90 treatment and 90 comparison) students. 

Notes: Statistical significance levels for two-sided tests are indicated with asterisks, as follows: *** = 1 percent; ** = 5 
percent; * = 10 percent. The difference is not statistically significant. 


English Learners 

We found that few English language learners remain enrolled in the focal schools 
through all five grades. Four of the five schools included in this analysis moved building 
during the period we studied, which likely exacerbated reductions in the number of 
students who were continuously enrolled. 


As a result, current or former English learners received a much lower dose of dual 
language than students never identified as English learners. Among students who were 
enrolled in 5th grade in one of the dual language treatment schools from 2014-2015 to 
2018-2019, 75 percent of students who were never English learner were enrolled in 
that school for 5 years or more compared to 38 percent of current or former English 
learners. The difference in average years of enrollment in dual language between current 
or former English learners and students never identified as English learners is statistically 
significant. Unfortunately, we do not know if this finding is similar or different from what 
occurs in traditional schools. 


Among the 441 current or former English learners who enrolled in the dual language 
schools over the follow up period, 80 were enrolled in the same dual language school 
in first and fifth grade — and 71 of those 80 English learners enrolled in the same dual 
language school in first and fifth grade had the test scores required for inclusion in 
analysis. This means the 7! dual language English learners included in the analysis are 
not typical of English learners in the dual language schools. In addition, the very small 
sample size means that error bands are wide, making it difficult to detect impacts. 
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For English learners, ELA achievement and the proportion of 
students redesignated fluent English proficient by 5th grade 
did not differ significantly between students enrolled in dual 
language schools from 1st to 5th grade and students enrolled 
in English language schools from 1st to 5th grade. 
e The average 5" grade ELA scores are higher for kindergarten English 
learners enrolled in dual language (equivalent to about half of typical annual 
gains in ELA) than English learners enrolled in non-dual language elementary 


schools; however, the error bands include small negative and large positive 
effects.” 

e = Ahigher proportion of kindergarten English learners enrolled in dual language 
from first to fifth grade were redesignated fluent English proficient than 
English learners enrolled in non-dual language elementary schools from first to 
fifth grade (41 vs 37 percent); however, the error bands suggest that dual 
language education could be associated with a small negative or large positive 
effect.” 


Exhibit 11. Dual Language Elementary School Matching Study 
Findings: Difference Between Treatment (Dual Language) and 
Comparison (Non-Dual Language) 
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Source(s): NM PED Administrative Data 

Sample: The sample includes 142 students (71 treatment and 71 comparison) students. 
Notes: Statistical significance levels for two-sided tests are indicated with asterisks, as follows: 
*** = 1 percent; ** = 5 percent; * = 10 percent. None of the impacts are statistically significant. 


22 These estimates suggest that kindergarten English learners in dual language education are no 
more than one-eighth of a of school year behind and up to one year ahead of similar English 
learners in non-dual language schools. 


23 Dual language could reduce the probability that kindergarten Englisher learners are 
redesignated by the end of 5 grade by 4 percentage points or could increase the probability 
by up to 42 percentage points. 
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Consideration of Prior Research 


This study was originally designed to replicate a rigorous dual language study — a lottery 
study -- conducted in Portland, Oregon. That study found a 0.13 (or 13% of a standard 
deviation) statistically significant positive effect on ELA performance across all students. 
For English learners, the study found that lottery admission reduced the probability of 
remaining classified as an English learner in 6th or 7th grade, a favorable finding 
equivalent to increasing the probability of being reclassified fluent English proficient. 
Although we do not find a statistically significant effect of dual language on 
reclassification, the point estimate is similar to the one found in the Portland study. 


Comparison to the Portland study is important because the Albuquerque context is 
dramatically different from the Portland context. For example, in the Portland sample, 
13% of students were ever English learners and 17% were Hispanic compared to 30% 
and 70% respectively in our analysis of all students. In Portland, four partner languages 
were offered (Spanish, Russian, Japanese, and Mandarin) across study schools and the 
authors do not report alignment between home language and these partner languages 
amongst English Learners. Although we do not have direct data on home language for 
students in our sample, 96% of English Learners in the sample are Hispanic and Spanish 
is the partner language at all five focal dual language schools. 


Observational studies have consistently found that learning another language while 
learning English does not hinder performance in English and developing literacy in two 
languages leads to higher metalinguistic awareness and cognitive strengths not found in 
monolingual peers (Bialystok, Craik, and Luk, 2012). Principals report that dual language 
students in this study have gained substantial skills in reading and writing in Spanish in 
addition to their English literacy skills. Further impact research should prioritize larger 
sample sizes, and the direct measurement of partner language proficiency as well as 
sociocultural competence, which is the third goal of dual language programs. 


Implementation Study Findings 


As a part of our implementation study, we conducted virtual focus groups with three 
of the five focal DL charter schools. Principal surveys provided additional background 
on the history and program structure.” Virtual site visits to DL schools led by Dual 
Language Education of New Mexico (DLeNM) provided perspectives and context from 
parents, teachers, and school leaders about student success and program sustainability. 


Four DL school leaders reported that their DL school programs were developed to 
respond to a specific community need and/or to offer parents higher-quality options 
than those previously available. Two principals specified that there was a community 
need for bilingual education and dual language programming in Albuquerque and that 
many traditional public school options were overcrowded. Parents, teachers, other 
community members, and non-profit organizations were founding members of the focal 
DL schools. 


The DL focal schools defined their DL program structure as one-way or two-way” and 
reported the percentage of content taught in each program language as 50/50, 80/20, 
90/10, or a variation thereof. Several other critical components separated them from 
other bilingual program models. As noted in the DL Logic Model (Exhibits 19 and 20 in 
English and Spanish respectively), DL programs ideally fulfill their mission by basing 


24 This section authored by our DL implementation team leads (Mr. Michael Rodriguez of Dual 
Language Education of New Mexico and Ms. Jacqueline Mendez of Abt Associates). 


25 “One-way” refers to foreign language immersion or developmental bilingual education in the 
southwestern United States; “Two-way” refers to enrolling students from two language 
groups, https://www.cal.org/twi/glossary.htm. 
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DL School Program Key Characteristics 


Strategic selection of instructional 
materials that support biliteracy 
development 


Clearly defined Curriculum Alignment 
Plans that demonstrate the vertical and 
horizontal alignment of what content is 
being taught in each language, for how 
long, at each grade level 


Structures for capacity building, language 
equity, and the allocation of resources 
are evident 


Linguistically and culturally diverse staff 


Planning and implementation of creative 
lessons that leverage cultural, linguistic, 
and community assets 


Instructional strategies that are 
implemented consistently 

School culture that celebrates cultural 
traditions and heritage 


Systems that ensure the consistency, 
alignment, fidelity, and intentional use of 
data in various languages 


Sustainability-ownership of program 


Support for school staff through ongoing 
professional development, collaboration, 
and resources 


Family and community engagement 


Consistent review of assessment data, 
and tailored intervention based on 
student needs 
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decisions on the fundamental goals of academic achievement, bilingualism and biliteracy, 
and sociocultural competence, as well as key actions that need to be intentionally 
planned. The leader at one school described its DL school environment: 


“[It] provides an equitable environment where students can share their 
backgrounds, and through their interactions they are able to take on learning a 
new language and sharing of cultures.” —DL Principal 


All DL focal schools implemented school-wide DL programs, rather than the more 
common model of DL “strands,” wherein a portion of the students at a school receive 
a dual language curriculum within an otherwise English-dominant school. Focus groups 
indicated that this school-wide model means there are never any instances where 
decisions are made for the school that can contradict or conflict with the foundational 
beliefs or mission of the DL program. In other words, all stakeholders are grounded in 
the same model, methodology, and instructional path to student success. School-wide 
DL programs are designed to influence the entire community and allow for systemic 
changes that focus on addressing racial, linguistic, cultural, and political issues. Focal DL 
school respondents described the various strengths and characteristics of their 
programs that flow from their school-wide approach to DL and shared vision (see side 
bar). 


Me encanta poder compartir de mis experiencias y mostrar un mundo diferente 
a mis estudiantes. [I love being able to share from my experiences and show my 
students a different world.] - DL Teacher 


Parents described their motivations for enrolling their child in the DL focal schools as 
being driven by the school’s focus on bilingualism and biliteracy, culture, and family. 
They noted that this was a benefit that was not provided to them by a traditional public 
education program. Many parents mentioned that their DL program provided their child 
with the ability to speak to grandparents or other family members in their native 
language. 


Tengo tres hijos... quiero que sean 100% bilingiies para que tengan un futuro 
mejor. [I have three children ... | want them to be 100% bilingual so that they 
have a better future.] -DL Parent 


Some parents also commented on the asset that the DL school program exposure 
provides for their child’s long-term career advancement and learning and appreciation 
of another language and culture. International festivals that focused on learning about 
other countries and cultures as well as Baile folklérico”® expanded families’ and students’ 
cultural experiences. Two parents described how the DL program is building meaningful 
cultural experiences for their children: 


Valoran mucho mi lengua natal y para mi es muy importante que mis (hijos) 
conozcan sus raices. [They [the DL school] value my native language very much 
and for me it is very important that my (children) know their roots.] -DL Parent 


“| appreciate an experience outside of a White-centric society. [My kids] have 
a plethora of other children who have totally different lives and experiences. 
There are teachers who have immigrated [to the United States] themselves, and 
hearing about their childhoods and other countries, | just think it’s special and 
unique and meaningful.” -DL Parent 


Parents generally stated that the school staff were committed to the school and 
engaging families in the school community. Parents highlighted high teacher retention as 
an extremely positive aspect of the school that demonstrated their dedication to the 
DL model. Parents described the regular school meetings with the principal as helpful 


26 This is a traditional Mexican dance that emphasizes local folk culture and ballet characteristics 
which also incorporates traditional Mexican dress. 
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DL School Program Key Challenges 


Reported 
The designation of school 
grading/effectiveness by the Public 


Education Department is English-centric 


There are currently no systems in New 
Mexico for charter schools to raise capital 
funds through bond elections in the same 
way as traditional districts 


With no clear neighborhood boundaries, 
extra effort is required in order to create a 
cohesive school community 


Balancing language profiles under the 
lottery / demographics of students served 
and how it might affect outcomes 


Charter schools in New Mexico can often 
feel isolated as they are not part of a larger 
management organization or supported 
by a district 
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and as providing families with direct access to the school leadership team to get updates, 
ask questions, and share their input. Parents also reported that there was equitable 
access to materials and communications were shared in both program languages. One 
parent with two daughters enrolled in a focal DL school said: 


“| feel that they really value Spanish at the school, and it shows. My kids went 
from not speaking any Spanish and learned so much. People are amazed that we 
don’t speak Spanish at home. She’s in Advanced Placement (AP) Spanish now. 
The main reason I chose this school for them was for them to learn Spanish and 
have the advantage of the second language.” —DL Parent 


The teachers reported various benefits of the DL school learning environment. Some 
examples included an emphasis on community involvement, the opportunity to work 
with a diverse school staff and students, varied professional development to ensure 
fidelity to the DL program model, the capacity to be creative with curriculum and 
professional development, and trust and support from the leadership team. For 
example, one teacher mentioned how they had a staff meeting where they discussed 
the Guiding Principles for Dual Language Education” and assessed where they were on 
each element. 


Lo riqueza del vocabulario que usamos con todo los estudiantes ensefia 
principalmente un lenguaje académico en todas las dreas (del contenido). [The 
richness of the vocabulary we use with all students primarily teaches the 
academic language in all (content) areas.] - DL Teacher 


“One of the blessings of this school that needs to stand out is that it’s not just the 
differentiation between English and Spanish, but we have an opportunity to ... 
exposure through our great, diverse teaching body... that we have a lot of 
representation from various countries (Mexico, Panama, Europe, Central/South 
America). We have in the diversity of our student and teacher body [something] 
that provides enrichment for all. The kids get to hear new vocabulary, and that 
brings greater linguistic awareness and diversity to language communities. I don’t 
see this at other schools that I’ve taught at.” - DL Teacher 


Teachers reported having a variety of opportunities for collaboration and decision- 
making at their DL school. Some examples include grade-level team meetings, 
professional learning committee meetings, unit and lesson development and 
modification, co-teaching, peer observations and sharing of strategies, and regular 
formal and informal professional development. Teachers explained that as a result, staff 
are much more aligned and focused on the same mission priorities. 


The DL focal schools reported challenges to their program structure and sustainability 
(see side bar). These challenges primarily related to English-centricity in educational 
systems and to isolation, and financial and enrollment constraints from the charter 
structure. On the other hand, focal DL charter schools reflected on their flexibility in 
decision-making and the ability to develop effective sustainable structures and make 
changes as needed. The school staff interviewed at these focal schools generally 
reported using their autonomy to make swift decisions and pivot in directions that are 
in the best interest of their mission, their diverse staff with shared goals, and the 
celebration of language and culture. 
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Charter Schools in the Study Sample 


In this chapter we provide an overview of the focal charter schools that participated in 
this study. Most of the data in this chapter comes from a survey of principals, focused on 
the elements of the logic model for the charter school theory of change. 


Study Sample flow from 97 charter schools to 21 participants 


The studies started with the full sample of 97 charter schools in New Mexico. Based on 
a survey of charter schools and expert consultation, we identified 30 schools across the 
state that received more applications than they had seats available and were eligible for 
the study. Of those 30 schools, 2! agreed to participate in this study. Ultimately, only 
ten oversubscribed schools provided lottery data that allowed us include them in the 
lottery study: i.e. the records allowed us to identify the students that were admitted in 
the initial lottery and follow the students from application through outcome 
measurement. All 21 schools were included the matching studies. Schools that served 
multiple grade bands (such as K-8 or middle-high schools) are included in multiple 
studies. 


Nearly half the focal charter schools in this study are located 
outside urban areas. 


The 21 focal schools were spread across the state, with almost half (48%) of schools 
located in small towns and rural areas and the remaining schools in larger urban and 
suburban areas. The five focal dual language charter schools and three additional charter 
schools were located in the Albuquerque metro area. Although the large proportion of 
schools in rural settings is not unusual for New Mexico, it is atypical in charter school 
research. This study offers important new information on the effectiveness of charters 
in locales where school choice is often limited to a single traditional public school or a 
single charter school. Earlier research on charter schools has focused on urban settings, 
where families routinely have multiple public school choice options. 


Characteristics of schools in this study vary as expected, 
given that they each were founded by local community 
groups. 


The charter schools that participated in this study represent a diversity of missions, 
strategies, and key educational components. This is expected because each charter 
school in New Mexico originates independently. This section provides background 
information on the characteristics of the schools in our study to help the reader 
understand the types of practices and programs offered by the focal schools. 


We surveyed school leaders to better understand how the key ingredients of an 
education at their school aligned with the logic model in Exhibit 18. Twenty of the 21 
schools (95%) responded to our survey,* allowing us to provide a summary of the key 
features that make these charter schools unique.” 


School Origin: All responding focal schools reported that they were founded by 
teachers, parents, and/or community members to respond to a perceived community 
need and/or demand from families for schools offering quality, rigor, proximity or a 
specific program-focus, such as dual language, STEM or meeting the needs of an under- 
served community. 
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Grades Served: Of the 21 schools in our study, fifteen cover multiple educational 
levels: eight K-7 or K-8 schools, five middle-high schools, and two K-12 schools. We 
only have six schools with more traditional grade spans: two elementary schools, one 
middle school and three high schools. As mentioned earlier, schools are included in our 
sub-studies based on the grades they serve. 


Nearly all schools reported 
that the majority of their 
teachers were veteran 


teachers. 


Principal and Teacher Tenure: More than half of the schools reported having had 
only one or two principals in the past 10 years; one school reported six principals. The 
average across focal schools was 2.5 principals in the past 10 years. 


And all but one school reported that two-thirds or more of their teachers are veteran 
teachers with more than four years of teaching experience. We cannot say whether this 
is more or less than control schools because we do not have data available to make that 
comparison. 


Instructional Time: Three of the 21 study schools had four-day school weeks; 
however, this is not unusual for charter schools in New Mexico, amongst which 20% 
have a four-day per week schedule.*" Instructional time is not a systematic key difference 
between our charter schools and traditional schools. Study schools were split between 
whether they offered more (12 schools) instructional hours or the same or fewer (9 
schools) instructional hours as the district in which they were housed. 


Restorative Practices and Socioemotional Learning Curricula: Almost all 
schools (approximately 90%) across grade levels reported using both restorative 
practices and socioemotional learning curricula. The latter was almost always (94%) 
offered to all students at the school in a whole-school integrated approach, whereas 
restorative practices were used as a whole-school integrated approach at 65% of schools. 
The remainder used a stand-alone disciplinary approach. 


Mission Area: The 2I charter schools that participated in this study represent a 
diversity of mission strategies and key components — this is expected because each 
charter school in New Mexico originates independently. 


Schools Serving Primarily High School Students: All eight schools that served primarily 
students in high school grades (either in a traditional middle/high school or high school- 
only structure) had a mission focused on college preparation (two via Early College High 
School). Half of the high school mission statements also focused on one or more of the - : 
following: leadership development, preparation for global citizenship, and/or integration total instructional hours 
or focus on science, technology, engineering, and math (STEM) (Exhibit 12). relative to the district in 
which they are housed. 


Focal schools varied 
considerably in whether 
they offered more or fewer 


Exhibit 12. Mission Focus of the Eight Schools Serving Primarily High 
School Students 


College preparation 100% 
Early college high school 25% 
Leadership development 25% 
Preparation for global citizenship 25% 
STEM integration/focus 25% 


Restorative justice/social justice RMA 


International Baccalaureate [RK 


Source(s): Principal Survey 

Sample: The sample includes the 8 schools that served primarily students in high school grades including 5 middle/high 
schools and 3 high schools (excludes K-12 schools). 

Note: Principals were able to report multiple mission areas; as a result, some schools are included in multiple categories 
above. Some principals did not respond to this question; for these cases, the research team coded missions from school 
websites. 
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Schools Serving Primarily Elementary and Middle School Students: The remaining |3 schools 
that served primarily elementary and middle school aged students most frequently 
described their mission as focused on “preparation for global citizenship” (n=8), serving 
as a “community school” (n=6), offering dual language education (n=5), and providing 
“arts integration/focus” (n=3) (Exhibit |3). 


Exhibit 13. Mission Focus of the 13 Schools Serving Primarily 
Elementary and/or Middle School Students 


Preparation for global citizenship 
Community school 

Dual language education 

Arts integration/focus 

Leadership development 
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Socioemotional well-being 15% 
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"No excuses" approach to learning 
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Project-based learning 
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Source(s): Principal Survey 

Sample: The sample includes 13 schools that served primary students in elementary and middle school grades (2 
elementary schools, 2 K-12 schools, 8 K-7 or K-8 schools, and 1 middle school) 

Notes: Principals were able to report multiple mission areas; as a result, some schools are included in multiple categories 
above. Some principals did not respond to this question; for these cases, the research team coded missions from school 
websites. 


Mission Centricity: All responding principals described their own professional values 
and priorities as very aligned with their school’s mission. Almost all also reported that 
the mission drove their decisions. However, there was more variation in whether that 
mission drove practice at the school, with three-quarters reporting a strong influence of 
the mission on practice and one-quarter reporting more moderate influence. 


Exhibit 14. Responses to Questions about Mission Centricity of 
Schools in This Study 


To what extent do you feel that 

your own professional values 

and priorities align with your 
school’s mission? 


How often does your school’s 
mission inform the decisions 
you make? 


To what extent does your 
school’s mission is driving 
practice at your school? 
Sometimes 


6% 


To some 
extent 


6% 


To a ver 
great extent 


19% 


Extremely 
aligned 


37% 


Very 
aligned 


63% 


To a great 
extent 


56% 


Source(s): Principal Survey 
Sample: The sample includes 16 of the 21 participating schools; the remaining 5 schools did not provide a response to 
these questions. 
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WHO IS SERVED BY CHARTER SCHOOLS IN 
NEW MEXICO? 


In this chapter we provide important context about the types of students served by 
public charter schools in New Mexico relative to the full state population. 


Charter schools included in this study serve fewer 
economically disadvantaged students than do either all 
charters or all public schools in New Mexico. 


We compared the demographic traits of the 21 focal schools, all 97 charter schools, 
and the 862 total public schools in the state to understand who is served by charter 
schools in New Mexico. The charter schools that participated in our study, on average, 
serve fewer economically disadvantaged and fewer minority students than do either the 
full population of charters or the full population of public schools in the state (Exhibit 
15). 


At the average focal charter school, 49% of students received Free or Reduced Price 
Lunch (FRPL) compared to 64% at the average charter school and 78% at the average 
public school in New Mexico. The average focal school also served substantially more 
White students and fewer Native American students than the average charter school 
or the average public school. On the other hand, the average focal school was generally 
similar to the average charter and public school in the state in its proportion of Hispanic 
students, English learners, and special education students. The trends are similar in 
direction at all three school grade levels, but the differences are more pronounced for 
elementary charter schools. 


Exhibit 15. Characteristics of the Average School, Comparing Focal 
Schools, All Charter Schools, and All Public Schools in the State 


MM focal schools (n=21) Wall charters (n=97) MMM state (n=862) 
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Source: Common Core of Data, 2017-18, New Mexico Report card summary from 2017-18. 
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This evidence spotlights that charter schools in New Mexico overall, and the focal 
charter schools to a greater extent, were disproportionately serving economically 
advantaged and White students in 2017-18. There are a number of possible explanations 
for this finding. 


First, because each charter school originated locally, it is possible that demand led to 
more schools being located in areas accessible to more privileged families and/or 
providing a mission that appeals to these families. Charter schools are not required to 
advertise their presence, so knowledge about these school options could vary notably 
across parents and families, possibly with especially limited awareness among non- 
English-speaking families. For example, we know anecdotally that there is substantial 
demand for select charter schools among more privileged college-educated and 
professional families in Albuquerque and Santa Fe. Because admission should be offered 
to the more privileged students in proportion to their incidence in the lottery, it is not 
surprising that student bodies in these schools include a higher proportion of privileged 
students than their proportion than the overall school population. 


Second, it is also possible that logistical barriers such as distance to the school and lack 
of transportation mean that economically disadvantaged families, even if offered a seat, 
cannot always take the seat when desired. Few charter schools offer buses, which could 
limit access to the schools for less privileged families living a distance from the school. 


Third, over time, the rules giving preference to siblings of current students in 
subsequent lotteries could contribute to maintaining the demographic characteristics 
of students attending a school. (This can occur in either direction, perpetuating the 
attendance of economically disadvantaged families or perpetuating the attendance of 
more privileged families.)*" 


Similarly, on average, DL charter elementary schools included 
in this study serve dramatically fewer economically 
disadvantaged students than do similar schools in the 
Albuquerque/Rio Rancho area. 


The focal elementary DL charter schools serve dramatically fewer economically 
disadvantaged students than all public elementary schools or all elementary charter 
schools that offer DL in the surrounding area (Albuquerque/Rio Rancho). At the 
average focal school, 35% of students received FRPL, compared to 71% at the average 
public elementary school and 89% at the average elementary charter school that offers 
DL (see Exhibit 16). The focal schools are notably different from the other groups of 
schools in the proportions of White students, English learners, and Special Education 
students, as well. 


PRE- 


Reminder: our study 
focuses on a time 
prior to the COVID 
pandemic when 
students where in 
schools. 
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Exhibit 16. Characteristics of the Average Elementary School, 
Comparing Focal DL Charter Schools, All Schools offering DL in the 
Albuquerque/Rio Rancho Area, and All Public Schools in the same 
Area 


HM focal schools (n=5) = {all DL schools (n=35) Mmmm All Schools (n=109) 
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Source: Common Core of Data, 2017-18, New Mexico Report card summary from 2017-18, New Mexico PED Bilingual 
Multicultural Education Programs Report 2017-18. 

Note: Many DL schools other than focal schools offer a DL track and a traditional track. In our analysis, we include all 
students in those schools, as the data source does not allow us to identify only students in the DL program at schools 
offering such a track. 


These dramatic differences for the DL population further illustrate the ways in which 
the types of students served at schools using lotteries could stem from the admissions 
lotteries used to determine enrollment. Similar to all charters, there appear to be 
inconsistencies in the recruitment efforts targeting diverse families across schools. 
Interviews with some focal school principals found that because charter laws and 
magnet school regulations in New Mexico do not allow for lotteries to select students 
by their dominant language, some principals have used targeted recruiting practices as 
a means of bringing greater linguistic and cultural diversity to the applicant pool. 
However, clearly these efforts are not sufficient at ensuring the school population 
matches the city population. 


Transportation also poses a related logistical barrier. In particular, some of the focal DL 
charter schools do not offer buses, which could limit access to the schools for 
economically disadvantaged families living a distance from the school. In fact, this is seen 
in the differences in the greater relative diversity in the types of students enrolled in 
the three schools that offer buses and/or are located in areas accessible to such families 
compared to the two schools without buses in less accessible areas. 


THEORY OF CHANGE LOGIC MODELS 
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In this chapter we present theory of change logic models for charter schools and for dual 
language education. The logic models presented here were developed collaboratively by the 
Abt research team and our partners.”’ It is important to note that this study was unable to 
measure a number of the elements of these logic models. However, we believe it is critical 
for the reader to understand the full theory of change and for us to acknowledge that ideally 
we would have measured all aspects of the logic model from the perspectives of all key actors 
(shown in rows in the logic models in Exhibits 18, 19 and 20). Unfortunately, limited resources 
often mean that researchers must limit the breadth of topics covered in their research. 


Exhibit 17 (below) provides a brief overview of the elements of each theory of change logic 
model in this chapter. These logic models should be read from left to right, starting with 
inputs, followed by precursors, core activities and facilitators, and finally outcomes. Though 
a logic model shows each element as discrete, due to the practicality of limited space, Exhibit 
17 shows that the elements (inputs, precursors, core activities and facilitators, and outcomes) 
overlap in reality. 


Exhibit 17. Understanding the Elements of Theory of Change Logic 
Models 


THEORY OF CHANGE LOGIC MODEL 
PRECURSORS 
| RS 


Processes, tools, actions, etc. that are 
intentional to bring about intended program 
changes or results 


INPUTS 


Resources/documents/etc 
needed to start 


Resulting 


improvements 
+ Proximal 
* Short-term 
Action steps when getting * Medium-term 
a program off the ground * Long-term 


We recommend that in addition to supporting this research project, these logic models be 
used to support related to practice, policy, and research. The two logic models are intended 
to provide a comprehensive and yet succinct overview for practitioners, policymakers, and 
researchers in the field. Practitioners could use them to plan for program development and 
implementation or to carry out a needs assessment. Policymakers could use them to guide 
decision-making and resource allocation within a district or state or at the national level (for 
the DL logic model, particularly around questions of language use that are less well 
understood or appreciated within a monolingual framework). Researchers could use them to 
contextualize their research (e.g., charter school researchers might want to focus on logic 
model elements that are currently un- or under-tested, such as the concept of mission 
centricity). 


27 These expert partners (Matthew Pahl of Public Charter Schools of New Mexico, Dr. Elizabeth 
Howard of the University of Connecticut, and Mr. Michael Rodriguez and Mr. David Rogers of 
Dual Language Education of New Mexico) have written the theory of change overviews provided 
below to ensure the reader hears directly from them about the theory of change. 
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Charter School Theory of Change?® 


In addition to the elements described in the overview above, the logic model for charter 
schools in New Mexico (Exhibit 15 at the end of this chapter) identifies actors (shown in 
rows), including the school’s leader, school staff, the students who attend the school, and 
their families. The logic model illustrates how charter schools are formed and how the 
autonomies given to charter schools provide better opportunities for students. 


It all starts with the state’s Charter Schools Act””, which identifies three primary elements 
of charters: 


|. Aschool that is independently governed by its governing council; 
2. A unique mission that is often rooted in a philosophy, goal, or content area; and 


3. Exemptions from certain portions of state law to provide freedom in how charter 
schools manage staff and execute in the classroom. 


These elements provide both purpose and autonomy for charter schools. The mission serves 
as the schools’ purpose. Independent governance and flexibilities in law provide autonomy. 
The governing board provides strategic guidance for that mission, which is then stewarded 
by the school leader and staff. This stewardship, labeled “Mission Fulfillment” in the exhibit, 
is at the core of what makes charter schools different and allows them to drive toward better 
student outcomes. 


It is the mission of the school — whether it is for all students to go to college, to implement 
a Montessori program, or to graduate students who had previously dropped out of high 
school — that is the lens through which the school makes decisions. The school’s leader and 
staff are selected on traditional teaching merits as well as their personal and professional 
alignment to the school’s mission. This results in higher staff retention. The staff as a whole 
is more invested in the mission, which guides the collective decisions of the board, school 
leader, and staff. Together a school will leverage its mission to make decisions about 
curriculum, electives and programs, professional development, and school culture. The 
execution of those decisions, which fulfills the mission, is what draws families and students 
to the schools. 


Families and students, because they chose the charter school and were not assigned to it, 
have higher investment in the school. The connection between engaged adults and families 
that choose the education offered at a charter school creates the potential for great 
outcomes. They are more invested in their school site, are more deeply engaged, and show 
better academic and life outcomes. This includes a higher sense of self-empowerment for 
students, higher rates of attendance, increased academic success, and increased graduation 
rates. Having attained higher-level academic skills, the student is poised to attend and 
complete college or start a fruitful career right out of high school. This success increases 
family well-being. 


Dual Language Education Theory of Change2? 


The logic model (Exhibit 16 in English and Exhibit 17 in Spanish)*° is anchored in the Guiding 
Principles for Dual Language Education” and supplemented by further research and best 


28 Written by Mr. Matthew Pahl, of Public Charter Schools of New Mexico. 


29 Written by Dr. Elizabeth Howard, of the University of Connecticut, with Mr. Michael Rodriguez 
and Mr. David Rogers, of Dual Language Education of New Mexico. 


30 Estamos encantados de apoyar la diseminacion de este informe a una audiencia de habla hispana. 
Pongase en contacto con Michael Rodriguez en michael@dlenm.org para obtener mas 
informacion. [We are happy to provide support for disseminating this report to a Spanish-speaking 
audience. Please contact Michael Rodriguez at michael@dlenm.org for more information.] 
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practice guidance in the field since the time of publication. In particular, there has been 
heightened attention to equity and social justice concerns within DL education in particular, ™ 
as well as a growing awareness of the urgency of culturally sustaining and anti-racist pedagogy 
in schools in general,” and this is reflected in the model. The logic model assumes that 
representatives from all stakeholder groups form a steering committee that oversees the 
enactment of the various phases, with equitable participation among all members, such that 
the point of view of the logic model is that of the steering committee rather than of the 
school personnel alone. 


Highlighted in red and flowing through the center as three interconnected, synchronous gears 
is a mapping sentence that aligns with the overarching input statement and articulates the 
goals, agents, and beliefs that consistently guide the progression of the inputs, precursors, 
and core activities and facilitators toward the desired outcomes. The mapping sentence calls 
for “mission fulfillment through a continuous focus on academic achievement, bilingualism 
and biliteracy, and sociocultural competence by the school, home, and community, through 
a stance that prioritizes equity and social justice, fosters critical consciousness, and is anti- 
racist and culturally and linguistically sustaining.” Similar to the input statement to which it is 
linked, this mapping sentence specifies the goals, partners, and dispositions required to attain 
the desired outcomes. 


Moving from left to right across the logic model, the first two columns on the left summarize 
the building blocks of program development and implementation. The inputs (in purple) are 
the resources that are required to initiate the program. In many ways, these inputs are 
common to any type of program, such as the selection of curricular materials and 
assessments, horizontal and vertical alignment of curriculum and instruction, and the 
development of a scope and sequence of the curriculum to guide instructional pacing. 
However, within a DL education program, the complexity of these inputs is much greater, as 
they must account for and be responsive to linguistic and cultural variation and serve to 
promote sociocultural competence and oral and written proficiency in both program 
languages, as well as academic achievement. All of this should be reflected in the strategic 
plan at the district level to ensure that there is system-wide understanding and commitment. 
In addition, the language allocation plan is a unique input that a monolingual program would 
not have. The second column (in green) introduces the precursors, or action steps, that are 
needed to get the program off the ground. For a DL program, this includes recruiting highly 
qualified administrators, support staff, and instructional personnel; enrolling students from 
varied language programs as articulated in the strategic plan; and engaging families and other 
community members and organizations as strategic partners. 


The middle of the logic model (in orange) conveys the core activities that the various actors 
undertake in order to achieve the desired outcomes. As with the inputs, many of the core 
activities are familiar to educators in any discipline, such as professional development, creating 
and modifying curricular materials, administering assessments, using effective instructional 
strategies, and promoting home-school-community engagement. Again, however, these 
activities are always informed by the goals and stance at the center of the model, meaning 
that they promote equity and social justice and affirm linguistic and cultural diversity; are 
specific to the priorities of DL stakeholders; and ultimately promote bilingualism, biliteracy, 
and sociocultural competence in addition to academic achievement. 


On that note, the final column (in blue) represents the four levels of intended outcomes, 
ranging from proximal to short-term to medium-term to long-term. These outcomes grow 
from shared responsibility and increased expectations to satisfaction and a sense of 
ownership. The equitable use of program languages in the school and the wider community 
increases metalinguistic awareness and the appreciation of linguistic diversity and fosters the 
development of leadership and shared advocacy. Ultimately the mission is realized through a 
sustained commitment to the vision and goals of the program and strengthened perspectives 
about linguistic and cultural diversity and the benefits of bilingualism. 
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Exhibit 18. Charter School Theory of Change Logic Model 
Charter School Theory of Change Logic Model 


This logic model was developed with support from Amold Ventures, Last updated 4/14/2020. Authors: Mathew Pahl, Public Charter Schools of New Mexico and Rachel McCormick, Abt Associates, 
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Exhibit 19. Dual Language Education Theory of Change Logic Model -English 


Dual si pe Education Theory of Change Logic Model UCONN Ge {av 


This logic model was developed with support from Arnold Veniures. Last updated 3/19/2021. Authors: Elizabeth Howard, Neag Scheol of Education University af Connecticut and Michael Rodriquez. Dual Language Education of New Maxico, in partnership with Jacqueline Mendaz and Rachel McCormick, Abt Associates. 
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THEORY OF CHANGE LOGIC MODELS 


Exhibit 20. Dual Language Education Theory of Change Logic Model -Spanish 


ACTIVIDADES CENTRALES Y FACILITADORES 


Teoria de cambio del modelo ldgico de la educacion de lenguaje dual 


Este modelo lgico fue desarrollado con el apoyo de Arnold Ventures. Ultima actualizacién 3/19/2021. Autores: Elizabeth Howard, Neag Schoo! of Education University of Connecticut y Michael Rodriguez, Dual Language Education of New Mexico, en asociacion con Jacqueline Méndez y Rachel McCormick, Abt Associates. 
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Notes 


NOTES 


'See Hussar et al, (2020). 
" See Steele et al. (2017). 


" The Yazzie/Martinez v. State of New Mexico lawsuit in 2018 ordered the state to “provide a 
sufficient education to all public school students,” especially Native students, English language 
learners, students from low-income families, and students with disabilities who would be 
irreparably harmed if the state did not act swiftly. Expansion of bilingual multicultural 
education programs has been recommended as a remedy, including by the New Mexico 
Legislative Finance Committee. 


“ We were unable to conduct a lottery study of dual language schools as planned due to 
limited availability of lottery data for these schools. 


” Known in the research literature as intention-to-treat randomized controlled trials. 


“ Known in the research literature as quasi-experimental designs with baseline propensity 
score matching. 


vl We refer to schools as elementary, middle, and high for ease. Note that these levels were 
defined by the grade a school served, and some schools are included in multiple levels (for 
example, because they are a Kindergarten through 8" grade school). 


“" See Gleason et al. (2010). 


* The Guiding Principles for Dual Language Education serves as a tool for planning, self-reflection, 
and continual improvement for DL programs and educators reflecting the research and 
knowledge, practices, and policies of dual language education. 


* Where possible, we include information on the non-respondents from the school’s web site 
and publicly available documents. Across items, response rates vary slightly--see Technical 
Appendix for details of item responses, missing responses, and data imputed by the research 
team. 


“The sample includes 20 charter schools and one dual language magnet school that admitted 
students by lottery. This magnet school exhibits many characteristics of charter schools 
identified in the charter school logic model, including the focus on mission, but is subject to 
administrative requirements that other schools in the sample are not. We include all 21 
schools in these summary statements and refer to the schools collectively as charter schools 
for the sake of simplicity. 


“See: New Mexico Legislative Finance Committee (2018). 


*"" Some schools also offer preferences to teachers’ students and the students of governing 
council members’ students which both limits seats available for the lottery and influences the 
demographic make-up of the school. 


*" The percentages of students on FRPL at each individual focal school is lower than the 
average for all DL schools and all APS/Rio Rancho elementary schools. 


x See New Mexico Administrative Code, 2007. 
*” See Howard et al. (2018). 


*' For example, see Cervantes-Soon et al. (2017); Flores (2021); Freire (2020); Palmer et al. 
(2019). 


“ For example, see Lyiscott (2019); Muhammad (2020); Paris & Alim (2017). 
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